Skip to content

Reference chains for surviving live-heap samples (PROF-15341)#644

Draft
jbachorik wants to merge 60 commits into
mainfrom
jb/reference-chains
Draft

Reference chains for surviving live-heap samples (PROF-15341)#644
jbachorik wants to merge 60 commits into
mainfrom
jb/reference-chains

Conversation

@jbachorik

@jbachorik jbachorik commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341): a JVMTI FollowReferences-based
BFS walk (ReferenceChainTracker) tags reachable objects and records
referrer-type chains back toward a GC root, bridged to
LivenessTracker::selectLeakCandidates()'s population-slope leak detection.

This revision also closes a structural gap found while adding end-to-end
coverage: a one-shot search could finish walking the whole reachable graph
before population-trend detection had accumulated enough GC epochs to flag a
candidate, making any object allocated after that point permanently
undiscoverable. Search restarts are now gated on an actual leak indication
existing (so a restarted walk is very likely to still find the object) plus
a PainBudget cooldown - a leaky bucket over cumulative safepoint time
spent, not a fixed interval, so a cheap search can restart sooner than an
expensive one.

Motivation:
PROF-15341 - give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive".

Additional Notes:
See doc/architecture/LiveHeapReferenceChains.md and
doc/architecture/LiveHeapReferenceChains-RemainingWorkPlan.md for the full
design history, open questions, and what's still provisional (per-pass
budget/hop-cap/TTL/frontier-cap/pain-budget-refill-rate defaults are all
round, unbenchmarked placeholders pending a future measurement pass).

How to test the change?:

  • ./gradlew :ddprof-lib:gtestDebug - full native unit suite, including new
    PainBudgetTest/SearchRestartTest coverage for the restart gate.
  • ./gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*" -
    genuinely separate-process end-to-end test: runs a real leaking-cache Java
    app in a child JVM and asserts the reconstructed chain's leaf class.
  • ./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*" -
    in-process coverage for the BFS engine, target-selection bridging, and
    abandonment reporting.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-15341

Unsure? Have a question? Request a review!

🤖 Generated with Claude Code

jbachorik and others added 3 commits July 8, 2026 17:35
…ALLOC_PERTURB_ for glibc test runs

Antagonist races tracer-style context propagation against virtual-thread
carrier churn to target the ContextStorageMode.THREAD stale-carrier
use-after-free. Since such heap corruption manifests silently until an
unrelated later allocation, also turn on glibc's own corruption checks
by default for chaos runs and ddprof-test suites (skipped on musl and
whenever a sanitizer/allocator already owns malloc via LD_PRELOAD).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
J9 hits a pre-existing, unrelated glibc heap-corruption bug that MALLOC_CHECK_ now surfaces as a hard abort; gate it off until that's investigated separately.
@jbachorik jbachorik added the AI label Jul 8, 2026
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #29515196421 | Commit: b8068c7 | Duration: 18m 12s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-07-16 16:42:51 UTC

jbachorik added 5 commits July 9, 2026 09:03
isRawPointer() only checked RAW_POINTER_MASK, letting raw J9 ASGCT BCIs
that never went through encode() misroute into JVMSupport::resolve(),
which asserts false for non-HotSpot VMs. Require ENCODED_MASK too.
It was registered in Main.java's antagonist factory but never added to
either config's ANTAGONISTS string, so it silently never ran in CI.
Include ACTIVE_PHASE_MILLIS in STALE_RACER_SLEEP_MILLIS so racers wake as
the next active phase begins instead of ~3s early. Bound stopGracefully's
total wait to the caller's timeout via a shared deadline instead of giving
each join/tryAcquire the full budget independently.
@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/125072101

@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit cfec595)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit: cfec5956f438cc43b62cd2c81cbe46077ef66168

⚠️ Significant outliers

  • 💥 akka-uct (JDK 25): latest crashed
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 25 💥 0 ms (0 iters) 💥 0 ms (0 iters) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / 1 1949 / 2055 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / · ✅ / · ✅ / · ✅ / · ✅ / · ✅ / ·
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 1 8727 / 8389 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8166 / 8396 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 2 1292 / 1256 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 1 1288 / 1290 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / 1 2830 / 3023 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2976 / 2821 ✅ / ✅ ✅ / ✅
naive-bayes 21 · / ✅ · / ✅ · / 3 · / 3555 · / ✅ · / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 3 3499 / 3485 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 1 1592 / 1697 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1922 / 1854 ✅ / ✅ ✅ / ✅

jbachorik and others added 3 commits July 9, 2026 18:10
…ntext via @trace

driverLoop() called JavaProfiler.getInstance() directly, but JavaProfiler is a
chaosCompileOnly dependency never bundled into chaos.jar and not visible to the
app classloader (the agent's copy is shaded/relocated). The resulting
NoClassDefFoundError is an Error, not an Exception, so it escaped the
catch (Exception e) block and silently killed the driver thread — leaving
activePhase stuck true and letting the sibling pinningChurnLoop run
full-throttle for the rest of the test instead of its intended
active/quiet duty cycle, which is the likely driver behind the OOM seen in
reliability-chaos-aarch64 [profiler, gmalloc, 21.0.3-tem].

Rewrite the antagonist to only touch the tracer through @Trace-annotated
methods (the same proven-safe dd-trace-api compileOnly pattern already used
by TraceContextAntagonist), letting the real tracer and its virtual-thread
instrumentation drive setContext/clearContext instead of calling
com.datadoghq.profiler.* directly. Drops the custom-attribute/baggage
simulation in favor of nested @trace hops plus forceUnmount and
stale-buffer racer threads that still race OtelContextStorage's
thread-scoped fallback across carrier churn.

Verified with a 90s live run (JDK 21.0.11, profiler+tracer, gmalloc): no
NoClassDefFoundError, no exceptions, completed cleanly with RC=0.

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without a RUNTIME arg, chaos_check.sh silently proceeds with an empty
value, which becomes --duration s in the chaos harness invocation and
fails 15+ seconds later — after the JDK/agent/jar setup — with a
NumberFormatException buried in the harness output, reported generically
as "FAIL:Chaos harness crashed (RC=1)". Check for a missing RUNTIME right
after arg parsing and fail immediately with a clear usage message.

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik added a commit that referenced this pull request Jul 10, 2026
…racking

Addresses review findings from PR #644 (reference chains for surviving
live-heap samples): FrontierTable's shared-lock mutation race, a
single-byte JFR event size prefix that silently truncates chains longer
than 255 bytes, blocking sample-lock retries in writeReferenceChain()
now bounded by a shared per-batch deadline with a drop counter,
unvalidated referencechains sub-option values, a startThread()/
pthread_kill() race publishing _running before the thread handle is
initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags()
now surfacing GetObjectsWithTags() failures so restartSearch() never
resets tag state prematurely, resolveLoadedClasses() skipping its
per-class scan only when the loaded-class count is unchanged (not just
non-decreasing), a spurious COMPLETED state after a failed first-pass
FollowReferences call, and removal of a leftover debug helper in
ExternalProcessReferenceChainTest. Adds regression test coverage for
release-failure and negative-value option paths.

Verified via the full ddprof-lib gtestDebug suite (149/149 tasks,
57/57 referenceChains_ut tests).

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 0714bd3)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit: 0714bd3166261f2710c22414dbfa68c11c699bdc

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -2.6% (2834→2761 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10184 ms (21 iters) ✅ 10371 ms (21 iters) ≈ +1.8% (±11.5%) — / —
akka-uct 25 ✅ 8891 ms (24 iters) ✅ 8865 ms (24 iters) ≈ -0.3% (±10.2%) — / —
finagle-chirper 21 ✅ 6003 ms (33 iters) ✅ 6034 ms (33 iters) ≈ +0.5% (±25.2%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5507 ms (36 iters) ✅ 5426 ms (36 iters) ≈ -1.5% (±23.8%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2834 ms (66 iters) ✅ 2761 ms (67 iters) 🟢 -2.6% — / —
fj-kmeans 25 ✅ 2824 ms (66 iters) ✅ 2838 ms (66 iters) ≈ +0.5% (±2.7%) — / —
future-genetic 21 ✅ 2113 ms (88 iters) ✅ 2065 ms (89 iters) ≈ -2.3% (±2.6%) — / —
future-genetic 25 ✅ 2059 ms (90 iters) ✅ 2102 ms (89 iters) ≈ +2.1% (±2.6%) — / —
naive-bayes 21 ✅ 1283 ms (134 iters) ✅ 1260 ms (135 iters) ≈ -1.8% (±32%) — / —
naive-bayes 25 ✅ 1016 ms (169 iters) ✅ 1017 ms (168 iters) ≈ +0.1% (±31.5%) — / —
reactors 21 ✅ 16658 ms (15 iters) ✅ 16371 ms (16 iters) ≈ -1.7% (±9%) — / —
reactors 25 ✅ 18507 ms (15 iters) ✅ 18056 ms (15 iters) ≈ -2.4% (±5.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 1 1929 / 2034 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 1 2310 / 2113 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 5 8529 / 8852 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 2 8635 / 8390 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 5 1284 / 1281 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 5 / 2 1283 / 1285 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 1 2930 / 2864 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2963 / 2953 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 7 / 6 3558 / 3510 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 4 3463 / 3469 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 1 1564 / 1782 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1905 / 1681 ✅ / ✅ ✅ / ✅

jbachorik and others added 3 commits July 10, 2026 21:08
Extract chaos_check.sh's logic into utils/run-chaos-harness.sh for local
repro. Add a #ifdef DEBUG-only watchdog in ProfiledThread's free path
(threadLocalData.cpp) that detects writes into freed OTel-context memory,
and beef up VirtualThreadContextCascadeAntagonist's stale-carrier racing
(continuous decoupled driver, randomized race window, dead-code removal).

Fixes from review: Main.java antagonist-start try/finally, watcher thread
signal-mask inheritance, racer-loop backoff under saturation, redundant
frame_ut.cpp assertions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
watchFreedContextMemory() deliberately read freed ProfiledThread memory
to detect stale writes, tripping ASan/TSan on every free (e.g. in
forced_unwind_ut) unrelated to the actual race under investigation.
@dd-octo-sts

dd-octo-sts Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 700d838)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit: 700d838fc33298d5789fd9e1b29c8a142c11f41e

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +4.5% (2598→2715 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10425 ms (21 iters) ✅ 10259 ms (21 iters) ≈ -1.6% (±11.1%) — / —
akka-uct 25 ✅ 8748 ms (24 iters) ✅ 8797 ms (24 iters) ≈ +0.6% (±10.5%) — / —
finagle-chirper 21 ✅ 5970 ms (33 iters) ✅ 5982 ms (33 iters) ≈ +0.2% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5469 ms (36 iters) ✅ 5468 ms (36 iters) ≈ -0% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2598 ms (73 iters) ✅ 2715 ms (69 iters) 🔴 +4.5% — / —
fj-kmeans 25 ✅ 2784 ms (67 iters) ✅ 2847 ms (66 iters) ≈ +2.3% (±2.7%) — / —
future-genetic 21 ✅ 2122 ms (87 iters) ✅ 2126 ms (88 iters) ≈ +0.2% (±2.7%) — / —
future-genetic 25 ✅ 2035 ms (91 iters) ✅ 2057 ms (91 iters) ≈ +1.1% (±2.7%) — / —
naive-bayes 21 ✅ 1304 ms (131 iters) ✅ 1253 ms (136 iters) ≈ -3.9% (±32.2%) — / —
naive-bayes 25 ✅ 1015 ms (169 iters) ✅ 1018 ms (168 iters) ≈ +0.3% (±31.8%) — / —
reactors 21 ✅ 16107 ms (15 iters) ✅ 16847 ms (15 iters) ≈ +4.6% (±7.9%) — / —
reactors 25 ✅ 18469 ms (15 iters) ✅ 19075 ms (14 iters) ≈ +3.3% (±4.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / ✅ 1950 / 1991 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 2 2228 / 2453 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 6 / 2 8686 / 8417 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8561 / 8290 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1272 / 1289 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1287 / 1298 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 2912 / 3017 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2907 / 2872 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 3 3515 / 3489 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 7 3498 / 3463 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 2 1732 / 1701 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1780 / 1924 ✅ / ✅ ✅ / ✅

jbachorik and others added 8 commits July 13, 2026 13:41
The skip's only justification was PROF-15360, closed as invalid in
favor of PROF-15364 (isRawPointer() false-positive, fixed by this PR).
No separate J9 heap-corruption issue is on record, so J9 now gets the
same MALLOC_CHECK_/MALLOC_PERTURB_ coverage as glibc/HotSpot.
…IER mode

Without --add-exports java.base/jdk.internal.misc=ALL-UNNAMED, OtelContextStorage
silently falls back to ContextStorageMode.THREAD, which vthread-context-cascade
was built to catch: virtual-thread context buffers pinned to a since-freed
carrier's ProfiledThread, corrupting the heap.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
profiler+tracer chaos runs OOMKilled on aarch64 (exit 137) with gmalloc/jemalloc
but not tcmalloc, which already had aggressive-decommit tuning for the same
thread-churn RSS-inflation issue. Apply the equivalent tuning to the other two
allocators.
Measured on matching-arch hosts: this workload floats at ~2.6-2.75GB RSS
regardless of MALLOC_CHECK_/allocator tuning, ruling that out as the OOM
cause. Trim -Xmx to 1536m on aarch64 for real headroom, and log the
container's actual cgroup memory limit so future OOMs are diagnosable
from the job log directly.
jbachorik and others added 21 commits July 16, 2026 13:29
Fixes a structural gap: a one-shot BFS search could finish walking the
whole reachable graph before LivenessTracker's population-trend
detection had enough epochs to flag a candidate, making later-created
leaking objects permanently undiscoverable.

- PainBudget (painBudget.h): leaky bucket over cost (ms), not event
  rate - gates how soon a restarted search may take its first pass.
- ReferenceChainTracker::canAffordNewSearch()/restartSearch(): a
  terminal search restarts once LivenessTracker reports a leak
  candidate and the pain budget has drained; unaffected when
  generations=true isn't set.
- FrontierTable::resetForRestart(), painbudget=N config sub-option
  (default 1%).
- gtest coverage for PainBudget and the restart gate; verified against
  ExternalProcessReferenceChainTest (separate-process leaking-cache
  scenario), which now passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ce-chain JFR roundtrip test

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…shared test-JVM heap

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…owth

_table_size was volatile but updated via __sync_bool_compare_and_swap,
which has no synchronizes-with edge under the C++ memory model - switch
to std::atomic<int> with relaxed ops. Also clamp the reference-chain
tests' per-round allocation growth at round 10 (trend-eligibility point)
so retry rounds stop adding unbounded garbage, which was causing OOME in
CI's shared test-JVM heap.
The glibc-aarch64 job never installed gtest/gmock, unlike amd64 and
musl jobs, so all native gtests silently skipped there. This surfaced
as a hard failure once ReferenceChainJfrParserTest started depending
on the roundtrip gtest actually producing its JFR file.
…from reference-chain BFS reconstruction

Both mechanisms are probabilistic and hard to reliably co-occur in one JVM run; new JNI test seams (guarded by #ifdef DEBUG) let JUnit tests seed population history and tag a known live object directly, verifying each end-to-end in isolation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…racking

Addresses review findings from PR #644 (reference chains for surviving
live-heap samples): FrontierTable's shared-lock mutation race, a
single-byte JFR event size prefix that silently truncates chains longer
than 255 bytes, blocking sample-lock retries in writeReferenceChain()
now bounded by a shared per-batch deadline with a drop counter,
unvalidated referencechains sub-option values, a startThread()/
pthread_kill() race publishing _running before the thread handle is
initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags()
now surfacing GetObjectsWithTags() failures so restartSearch() never
resets tag state prematurely, resolveLoadedClasses() skipping its
per-class scan only when the loaded-class count is unchanged (not just
non-decreasing), a spurious COMPLETED state after a failed first-pass
FollowReferences call, and removal of a leftover debug helper in
ExternalProcessReferenceChainTest. Adds regression test coverage for
release-failure and negative-value option paths.

Verified via the full ddprof-lib gtestDebug suite (149/149 tasks,
57/57 referenceChains_ut tests).

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
resetSearchStateForTest() and klassPopulationResetForTest() now stop the
BFS thread / take _table_lock respectively before mutating state the
thread otherwise owns exclusively. Also reset LivenessTracker's own
population history before shouldReconstructReferrerChainThroughUnboundedCacheLeak()
so a stale candidate from the prior test can't outrank CachedPayload, and
align its per-round growth rate with the sibling test's.
StringDictionary::clearAll() (on profiler restart) wipes dict ids but
JVMTI class tags survive, so ReferenceChainTracker::_class_tags held
ids into a namespace that no longer existed. Add a generation counter
to StringDictionary and have resolveLoadedClasses() detect a mismatch
and force re-resolution, reusing existing JVMTI tags instead of always
minting new ones. Also adds a firstpassbudget override for the
root-seeded first BFS pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…samples

cleanup_table()'s per-klass population accounting was gated on !forced,
skipping it entirely whenever track()'s table-overflow branch triggered a
forced sweep. Gate on a new is_epoch_owner check instead (true once per
genuinely new GC epoch, forced or not), so a forced sweep still folds a
sample using an already-cached klass id. resolveKlassId() (a real
Class.getName() Java upcall) still only runs on the organic path - that
part was a legitimate hot-path cost concern, just misdescribed as a JVMTI
safety requirement.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
resetSearchStateForTest() only reset the frontier table's occupancy,
not its capacity, so an earlier test's smaller framecap= permanently
capped the singleton for every later test in the same JVM.
…ceChainTestSeamsTest

An earlier test can leave the shared FrontierTable full/tiny
(shouldReportAbandonedSearchOnTinyFrontierCap deliberately caps it at 1);
without resetting first, tagAsReferenceChainRoot0()'s insert() fails
deterministically.
Profiler::dump() discarded _jfr.dump()'s Error entirely, and
OS::copyFile()'s sendfile() loop broke silently on short/failed
copies - both are debug-only visibility gaps that would otherwise
leave a lost reference-chain write or truncated chunk copy
unexplained in CI logs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same debug-only seeded-representative short-circuit already used by
ReferenceChainTrackingTest, ported to the external-process scenario:
propagates ddprof_test.config to the child JVM so it can bypass
LivenessTracker's probabilistic slope detection instead of depending
on it to notice CachedPayload within the fixed round/timeout budget.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ubuntu.com

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jbachorik jbachorik force-pushed the jb/reference-chains branch from 1299316 to baba070 Compare July 16, 2026 11:31
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit baba070)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: baba0702198bb700cacfe9ab28c900eaf7d3f729

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10403 ms (21 iters) ✅ 10169 ms (21 iters) ≈ -2.2% (±10.2%) — / —
akka-uct 25 ✅ 8907 ms (24 iters) ✅ 8843 ms (24 iters) ≈ -0.7% (±10.4%) — / —
finagle-chirper 21 ✅ 5989 ms (33 iters) ✅ 5996 ms (33 iters) ≈ +0.1% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5542 ms (36 iters) ✅ 5550 ms (36 iters) ≈ +0.1% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2791 ms (67 iters) ✅ 2838 ms (66 iters) ≈ +1.7% (±2.7%) — / —
fj-kmeans 25 ✅ 2852 ms (66 iters) ✅ 2834 ms (66 iters) ≈ -0.6% (±2.6%) — / —
future-genetic 21 ✅ 2086 ms (89 iters) ✅ 2064 ms (90 iters) ≈ -1.1% (±2.6%) — / —
future-genetic 25 ✅ 2100 ms (88 iters) ✅ 2103 ms (88 iters) ≈ +0.1% (±2.5%) — / —
naive-bayes 21 ✅ 1266 ms (135 iters) ✅ 1263 ms (135 iters) ≈ -0.2% (±32.4%) — / —
naive-bayes 25 ✅ 1016 ms (168 iters) ✅ 1017 ms (168 iters) ≈ +0.1% (±31.8%) — / —
reactors 21 ✅ 16362 ms (15 iters) ✅ 16222 ms (15 iters) ≈ -0.9% (±8.7%) — / —
reactors 25 ✅ 18867 ms (15 iters) ✅ 19004 ms (15 iters) ≈ +0.7% (±4.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 1 1936 / 1969 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2258 / 2003 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 5 8622 / 8930 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 2 8497 / 8401 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 2 1283 / 1288 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1282 / 1292 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 4 / 1 2933 / 3028 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 3 2856 / 2850 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 4 3539 / 3501 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 7 / 5 3446 / 3450 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1682 / 1519 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1907 / 1977 ✅ / ✅ ✅ / ✅

… runs

ObjectSampler::start()/stop() gated calls into LivenessTracker on its own
(freshly correct) flags, but LivenessTracker never got a chance to refresh
its own _gc_generations from a run that requested neither liveness nor
generations tracking - leaving it stuck at whatever the previous test in
the same JVM last set it to.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 35c49c2)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit: 35c49c2663a2e6e270a0287e4e059b8dab047faa

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 25): runtime -3% (2842→2756 ms)
  • 🟢 future-genetic (JDK 21): runtime -2.9% (2142→2080 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10367 ms (21 iters) ✅ 10371 ms (21 iters) ≈ +0% (±11.3%) — / —
akka-uct 25 ✅ 8917 ms (24 iters) ✅ 8766 ms (24 iters) ≈ -1.7% (±10.2%) — / —
finagle-chirper 21 ✅ 5986 ms (33 iters) ✅ 5972 ms (33 iters) ≈ -0.2% (±25.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2801 ms (67 iters) ✅ 2750 ms (68 iters) ≈ -1.8% (±2.6%) — / —
fj-kmeans 25 ✅ 2842 ms (66 iters) ✅ 2756 ms (68 iters) 🟢 -3% — / —
future-genetic 21 ✅ 2142 ms (87 iters) ✅ 2080 ms (89 iters) 🟢 -2.9% — / —
future-genetic 25 ✅ 1991 ms (94 iters) ✅ 2035 ms (91 iters) ≈ +2.2% (±2.7%) — / —
naive-bayes 25 ✅ 1025 ms (167 iters) ✅ 1010 ms (169 iters) ≈ -1.5% (±31.4%) — / —
reactors 21 ✅ 16244 ms (15 iters) ✅ 16474 ms (15 iters) ≈ +1.4% (±7.9%) — / —
reactors 25 ✅ 18288 ms (15 iters) ✅ 18332 ms (15 iters) ≈ +0.2% (±3.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 4 1950 / 1991 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2303 / 2180 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 6 8767 / 8881 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 1 8034 / 8300 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 1 1316 / 1287 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 2 1289 / 1282 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 2889 / 2882 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 3 / 1 2935 / 2838 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 8 / 2 3496 / 3495 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 4 3477 / 3510 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1642 / 1586 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 3 1802 / 1759 ✅ / ✅ ✅ / ✅

…GcRoot

The one-shot root-seeded BFS walk can fire before round 1's addAll() in
this shared, no-forkEvery JVM, freezing gcRootHolder's children as
permanently empty. Seed one live ChainLink before resetting the search
so the backing array is never empty when the walk snapshots it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit a9fb05f)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit: a9fb05fe90d68c1097e48893b1256661ecdfce30

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -3.1% (2772→2687 ms)
  • 💥 naive-bayes (JDK 25): latest crashed
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10309 ms (21 iters) ✅ 10453 ms (21 iters) ≈ +1.4% (±11%) — / —
akka-uct 25 ✅ 8814 ms (24 iters) ✅ 8779 ms (24 iters) ≈ -0.4% (±10.4%) — / —
finagle-chirper 21 ✅ 6006 ms (33 iters) ✅ 6047 ms (33 iters) ≈ +0.7% (±25.8%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5467 ms (36 iters) ✅ 5455 ms (36 iters) ≈ -0.2% (±24.8%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2772 ms (68 iters) ✅ 2687 ms (70 iters) 🟢 -3.1% — / —
fj-kmeans 25 ✅ 2810 ms (66 iters) ✅ 2801 ms (66 iters) ≈ -0.3% (±2.6%) — / —
future-genetic 21 ✅ 2087 ms (88 iters) ✅ 2055 ms (90 iters) ≈ -1.5% (±2.7%) — / —
future-genetic 25 ✅ 2088 ms (89 iters) ✅ 2046 ms (91 iters) ≈ -2% (±2.5%) — / —
naive-bayes 21 ✅ 1233 ms (139 iters) ✅ 1257 ms (136 iters) ≈ +1.9% (±32.9%) — / —
naive-bayes 25 💥 0 ms (0 iters) 💥 0 ms (0 iters) — / —
reactors 21 ✅ 16947 ms (15 iters) ✅ 15936 ms (15 iters) ≈ -6% (±8.2%) — / —
reactors 25 ✅ 18681 ms (15 iters) ✅ 18731 ms (15 iters) ≈ +0.3% (±5.3%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 2008 / 2140 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2058 / 2262 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 3 8437 / 8783 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 3 8064 / 8228 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 6 / ✅ 1275 / 1291 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 2 1250 / 1261 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 2 2876 / 3036 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2912 / 2869 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 4 3492 / 3491 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1892 / 1633 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / 2 1914 / 1937 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 2033006)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit: 20330063bf66bf581cf9ae47f94cff709e5aa6be

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 25): runtime -3.7% (2848→2743 ms)
  • 🔴 future-genetic (JDK 25): runtime +3.2% (1994→2057 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10363 ms (21 iters) ✅ 10147 ms (21 iters) ≈ -2.1% (±11.6%) — / —
akka-uct 25 ✅ 8966 ms (24 iters) ✅ 8873 ms (24 iters) ≈ -1% (±10.2%) — / —
finagle-chirper 21 ✅ 6007 ms (33 iters) ✅ 5986 ms (33 iters) ≈ -0.3% (±26.1%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5488 ms (36 iters) ✅ 5433 ms (36 iters) ≈ -1% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2726 ms (68 iters) ✅ 2760 ms (68 iters) ≈ +1.2% (±2.6%) — / —
fj-kmeans 25 ✅ 2848 ms (66 iters) ✅ 2743 ms (69 iters) 🟢 -3.7% — / —
future-genetic 21 ✅ 2062 ms (90 iters) ✅ 2047 ms (90 iters) ≈ -0.7% (±2.6%) — / —
future-genetic 25 ✅ 1994 ms (93 iters) ✅ 2057 ms (90 iters) 🔴 +3.2% — / —
naive-bayes 21 ✅ 1269 ms (135 iters) ✅ 1258 ms (136 iters) ≈ -0.9% (±32.5%) — / —
naive-bayes 25 ✅ 1012 ms (169 iters) ✅ 1016 ms (168 iters) ≈ +0.4% (±31.6%) — / —
reactors 21 ✅ 16554 ms (15 iters) ✅ 16630 ms (15 iters) ≈ +0.5% (±8.5%) — / —
reactors 25 ✅ 18407 ms (15 iters) ✅ 18308 ms (15 iters) ≈ -0.5% (±4.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 2 2101 / 2026 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 3 2336 / 2365 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8772 / 8497 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8397 / 8470 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 1 1260 / 1274 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 3 1303 / 1307 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 2 2893 / 2882 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 2 2958 / 2912 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / 1 3540 / 3527 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 5 / 6 3458 / 3485 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1736 / 1754 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1990 / 1872 ✅ / ✅ ✅ / ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants